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(54) TiUe: MOTION VECTOR ESTIMATION AND DETECTION OF COVEREDAJNCOVERED IMAGE PARTS 
(57) Abstract 



D? 



D? 



In a method of detecting 
covered and uncovered parts in an 
image to be interpolated l)etween 
neighboring previous (I) and 
next (III) input images, backward 
motion vectors from the next 
input image to the previous input 
image, and having corresponding 
backward estimation errors, and 
forward motion vectors from 
the previous input image to the 
next input image, and having 
corresponding forward estimation 
errors, are determined, uncovued 
and covered parts arc detected 
in the neighboring previous and 
next input images, respectively, in 
the thus detected uncovered parts, 
uncovered parts in the image to 
be interpolated are detected by 

determining second backward esdmation errors by comparing both neighboring previous and next input images when partially shifted over 
the backward motion vectors to a temporal location of said image to be interpolated, and by comparing the second backward estimation 
errors to a threshold, and in the thus detected covered parts, covered parts in the image to be interpolated are detected by determining 
second forward estimation errors by comparing both neighboring previous and next input images when partially shifted over the forward 
motion vectors to a temporal location of the image to be interpolated, and by comparing the second forward estimation errors to the 
threshold. 



























— j — 




































1. 






















y 

f. 1 


































— y 














— , 




















i 








n 










-^- 




















lb 
















mdin 

ii ' 




























5- 
































































/u / r 
















V 
































Xr 
































— 1 ! 

1 i i 1 f 









Df 



t <WO 992252QA2_I_> 



FOR THE PURPOSES OF INFORMATION ONLY 



Codes used to identify States party to the PGT on the front pages of pamphlets publishing international applications under the PCX. 



AL 


Albania 


ES 


Spain 


LS 


Lesotho 


SI 


Slovmia 


AM 


Annenia 


FI 


.Fmland 


LT 


Lithuania 


SK 


Slovakia 


AT 


Austria 


FR 


Ranee 


LU 


Luxembourg 


SN 


Senegal . 


AU 


Australia 


GA 


Gabon 


LV 


Latvia 


SZ 


Swaziland 


AZ 


Azeibaijan 


GB 


United Kingdom 


MC 


Monaco 


TD 


Chad 


BA 


Bosnia and Herzegovina 


GE 


Georgia. 


MD 


Republic of Moldova 


TG 


Togo 


BB 


Barbados 


GH 


Ghana 


MG 


Madagascar 


TJ 


Tajikistan 


BE 


Belgium 


GN 


Guinea 


MK 


The former Yugoslav 


TM 


Turkmenistan 


BF 


Buitina Faso 


GR 


Greece 




Republic of Macedonia 


TR 


Turkey 


BG 


Bulgaria 


HU 


Hungary 


ML 


Mali 


TT 


Trinidad and Tobago 


BJ 


Benin 


IE 


Ireland 


MN 


Mongolia 


UA 


Ukraine 


BR 


Brazil 


IL 


Israel 


MR 


Mauritan^ 


UG 


Uganda 


BY 


Bebrus 


IS 


Iceland 


MW 


Malawi 


US 


United States of America 


CA 


Canada 


IT 


Italy 


MX 


Mexico 


UZ 


Uzbekistan 


CF 


Central African Republic 


JP 


Japan 


NE 


Niger 


VN 


Viet Nam 


CC 


Congo 


K£ 


Kenya 


NL 


Netherlands 


YU 


Yugoslavia 


CH 


Switzeriand 


KG 


Kyigyzstan 


NO 


Norway 


zw 


Zimbabwe 


CI 


C6tc d*Ivoirc 


KP 


Democratic I^BOp1e*s 


NZ 


New Zealand 






CM 


Cameroon 




Republic of Kotea 


PL 


Poland 






CN 


China 


KR 


Republic of Korea 


FT 


Portngal 






CU 


Cuba 


KZ 


Kazakstan 


RO 


Romania 






CZ 


Czedi Republic 


LC 


Saint Lttcia 


RU 


Russian Federation 






DE 


Germany 


U 


Liedi^istein 


SD 


Sudan 






DK 


Dcninark 


LK 


Sri Lanka 


SE 


Sweden 






EE 


Estonia 


LR 


Liberia 


SG 


Smgapore 







BMSOOCID: <WO 992252QA2J_> 



wo 99/22520 PCT/IB98/01594 

1 

Motion vector estimation and detection of covered/uncovered image parts. 



The invention relates to motion vector estimation and detection of 
covered/uncovered image pans. The invention also relates to motion-compensated 
interpolation, and to a display apparatus comprising means for carrying out such a motion- 
compensated interpolation. . 

5 

The paper "Motion estimation method using the spatio-temporal 
characteristics of moving objects", presented at the International Workshop on HDTV, Los 
Angeles, October 1996, by H. Sonehara et al., discloses a motion vector detection method 

10 which takes account of the spatio-temporal continuity of motion vectors and the local 

characteristics of pictures. Motion vector trajectories are traced to assess the spatio-temporal 
continuity of motion vectors. To remove erroneous motion vectors detected in the 
covered/uncovered background of the contour area of moving objects, motion vectors are 
estimated by means of a detection-area hierarchy and the correlation of luminance signals. It 

15 is necessary to identify, in two sequential fields, the area (covered and uncovered background 
area) in which the picture correlation becomes indeterminate. For this, a detection block is 
set in the current field and vectors are detected in relation to the next field. Then a detection 
block is set in the next field and vectors are detected in relation to the current field. In short, 
motion vectors are detected two-way. 

20 EP-B-0,535,066 discloses a method of assigning the appropriate one (if 

any) of a plurality of possible motion vectors to pixels of a video signal consisting of a series 
of pictures, the method comprising the steps of: a backwards assignment operation in which 
a first input picture is shifted by each of a plurality of possible vectors and compared to a 
second, succeeding picmre to determine assignment errors for each vector at each pixel; a 

25 forward assignment operation in which the second picture is shifted by each of a plurality of 
possible vectors and compared to the first picmre to determine assignment errors for each 
vector at each pixel; and using for each pixel the assignment errors determined in both 
forward assignment and backwards assignment operations to determine whether any of the 
vectors is appropriate for that pixel. The method may further comprise comparing the 
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assignment errors for the forward and backwards assignment operations for a pixel to 
determine whether the pixel may relate to revealed or obscured background. This combined 
vector assignment flags regions of both revealed and obscured background as 'no match' so 
that a fallback interpolation technique can be used. This fallback interpolation technique has 
5 not been described. In an alternative method, vectors from the instant of an input picture are 
converted to an output picmre at a different instant in time by projecting the input vectors 
forward in time (allowing for their motion). Several problems are inherent in projecting 
vectors forward in time. While these problems are not insuperable the technique of combined 
forward and backward assignment is simpler and therefore more attractive. 

10 

It is. inter alia, an object of the invention to provide a reliable motion 
detection / motion-compensated interpolation. To this end, a first aspect of the invention 
provides a motion vector estimation method and device as defined in claims 1 and 5. A 
15 second aspect of the invention provides an interpolation method as defined in claim 4. A 
third aspect of the invention provides a display apparams as defined in claim 6. A fourth 
aspect of the invention provides a method of and device for detecting covered and uncovered 
parts in an image to be interpolated as defined in claim 7 and 9. A fifth aspect of the 
invention provides a motion vector estimation device as defined by Claim 10. Advantageous 
20 embodiments are defined in the dependent claims. 

A primary aspect of the invention provides a method of detecting covered 
and uncovered pans in an image to be interpolated between neighboring previous and next 
input images, in which backward motion vectors from the next input image to the previous 
input image, and having corresponding backward estimation errors, and forward motion 
25 vectors from the previous input image to the next input image, and having corresponding 
forward estimation errors, are determined, uncovered and covered parts are detected in the 
neighboring previous and next input images, respectively, in the thus detected uncovered 
parts, uncovered parts in the image to be interpolated are detected by determining second 
backward estimation errors by comparing both neighboring previous and next input images 
30 when panially shifted over the backward motion vectors to a temporal location of said image 
to be interpolated, and by comparing the second backward estimation errors to a threshold, 
and in the thus detected covered parts, covered parts in the image to be interpolated are 
detected by determining second forward estimation errors by comparing both neighboring 
previous and next input images when partially shifted over the forward motion vectors to a 
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temporal location of the image to be interpolated, and by comparing the second forward 
estimation errors to the threshold. 

Another aspect of the invention provides a method of estimating motion 
vectors for an image to be interpolated between neighboring previous and next input images, 

5 in which motion from the next input image to the previous input image is detected to obtain 
backward motion vectors having corresponding backward estimation errors, motion from the 
previous input image to the next input image is detected to obtain forward motion vectors 
having corresponding forward estimation errors, uncovered and covered parts are detected in 
the image to be interpolated, motion vectors are furnished for the uncovered parts in 

10 dependence upon the forward motion vectors taken from locations wholly and/or partly 
shifted over the backward motion vector, and motion vectors for the covered parts are 
' furnished in dependence upon the backward motion vectors taken from locations wholly 
and/or partly shifted over the forward motion vectors. 

These and other aspects of the invention will be apparent from and 

15 elucidated with reference to the embodiments described hereinafter. 



In the drawings: 

Fig. 1 illustrates covering and uncovering background situations; 
20 Fig. 2 illustrates where in the interpolated image covering and uncovering 

occurs; and 

Fig. 3 shows a preferred embodiment of the current invention. 



25 A primary aspect of the present invention aims at a strong reduction, if 

not an elimination, of halo effects for systems that have access to both neighboring fields by 
analyzing in which of the two fields the information, that has to be interpolated, is present 
(i.e. not covered) using two motion vectors per location. This information is used to 
interpolate the motion compensated picmre from both neighboring images ignoring pixels 

30 from either of the two images that were either covered, or uncovered. The use of two motion 
estimators in parallel is considered to be realistic, even in consumer products, since earlier 
work [7, 8] has reduced the cost of a motion estimator to an affordable level. Nevertheless, 
an alternative will be discussed in which the additional calculations are only required in areas 
where covering or uncovering is expected, i.e. near picmre parts in which discontinuities 
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(contours) in the vector field occur. 

Basic requirement for the current algorithm is the availability of two 

motion vectors for every location. The first vector j5, is the one calculated by shifting a 
previous field to optimally match a current field, whereas the second is obtained by shifting 

5 the data firom the current field to optimally match a previous field. The required 

segmentation of the image in "normal, "covering", and "uncovering" regions will be dealt 
with later. Furst. we shall elucidate the motion vector calculation using the ftill-search 
block-matching motion estimator to calculate the motion vectors. This algorithm is not a very 
cost effective approach to the motion estimation problem, but the more effective ones are 

10 based on the same principle. 

In block-matching motion estimation algorithms, a displacement vector is 
assigned to the center X = (Xx, Xy)"^ of a block of pixels B(X) in the current field n by 
searching a similar block within a search area SA(X), also centered at X. but in the previous 
15 field n-1. The similar block has a center, which is shifted with respect to X over the 
displacement vector n). To find n), a number of candidate vectors C are 

evaluated applying an error measure (C. X. n) to quantify block similarity. More formally. 
CS*™* is defined as the set of candidates C , describing all possible (usually integer) 
displacements with respect to X within the search area SA(X) in the previous image: 

20 

CS"^ = { C I -N ^ C, :< N, -M ^ Cy ^ M} (1) 

where N and M are constants limiting SA(X). Furthermore, a block B(X) centered at X and 
of size X by Y consisting of pixel positions x = (x,y)^ in the present field n, is now 
25 considered: 

B(X) = I X, - X/2 ^ X ^ X, + X/2 A X^ - Y/2 ^ y ^ X^ + Y/2} (2) 
The displacement vector D(X, n) resulting from the block-matching 
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process, is a candidate vector C which yields the mihunum value of an error function 
(C, X, n): 

D{X, n) e {C € CS™ I €(Cj, n) j< €(K.Jf , n) VK G CS"^} (3) 

If, which is the common case, the vector D(x, n) with the smallest 
matching error is assigned to all pixel positions x in the block B(X): 

Vx 6 B(X): J5(jc, n) = jD(X, n) (4) 



rather than to the center pixel only, a large reduction of computations is achieved. 

As an implication, consecutive blocks B(X) are not overlapping. The 

error value for a given candidate vector is a function of the luminance values of the 
pixels in the current block and those of the shifted block from a previous field, summed over 

15 the block B(X), A common choice, which we will use, is the sum of the absolute 
differences: 



20 although a mean square error or a normalized cross correlation function are sometimes 
proposed. 

The difference between the first and the second motion estimator, using 
candidates C5 and C, respectively and resulting in motion vectors and Df, is that we 
propose in the method according to the invention, lies in the error criterion of (5). According 
25 to (5), the current field is shifted to optimally match the previous field (backward 

prediction). The second estimator that we propose is identical to the first, apart from (5) 
which is replaced by: 
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€^(C, X, n)= E \F{x, ny - Fix ' Cj, n-D] (6) 



This implies that is optimally suited for forward prediction. With two 
estimators defined above, a high match error of the best matching candidate means that it is: 
5 • difficult to predict the current field with the previous, for the forward predicting 

estimator applying (6), or that it is: 
• difficult to predict the previous field with the present, for the backwards 

predicting estimator applying (5). 

10 The first situation occurs in case of uncovering background, the other is 

typical for a covering of the background. The relative performance RP of the two motion 
estimators is now proposed to decide upon covering and uncovering: 

(7) 

(e^ + ep + 6 

and: 

f covering, (.RP^Th^) ^ 
15 decision ^ ^^^^^^^^^.^^^ W 



where 5 is introduced to prevent decisions of covering/uncovering in case of small match 
errors. In the experiments 6 was set to approximately 1 percent of the maximimi error. 

20 However, preferably the following decision is taken: 

decision . H-""""*- • ™;:™J> , (7/8a) 



I covering, ie^^-e^ * l/Th^+Th^) 



where Th, is introduced to prevent decisions of covering/uncovering in case of small match 
errors. In the experiments Th, was set to approximately 3 and Thz to 20. 

25 

When a block is detected as belonging to either the covering or the 
uncovering region, the 8 blocks surronding it are also assigned to that region, in order to 
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Fig. 1 illustrates the covering and uncovering background situations. A 
moving object, a circle, is shown moving (with motion vector D) from left to right. N 
.5 indicates the position in field n, while N + 1 indicates the position in field n + 1. Region I is 
uncovered in field n + 1, while for region III covering occurs. Region H can be interpolated 
from both fields. 

Although this brings some clarity about areas where covering and 
uncovering appears, the precision of the method is not optimally suited for the image(s) tcrbe 
10 interpolated. In that image only part of region I is uncovered and only part of region III 
becomes covered. The fraction depends on the temporal position of the interpolated image 
relative to the neighboring original images. If the interpolated image has to be valid at the 
temporal instance exactly in between the previous and next original, than the covering and 
uncovering areas are half the size that we detected. 

15 

Fig. 2 illustrates this case into more detail, and shows, particularly, 
regions in which covering and uncovering occurs, and where in the interpolated image which 
motion vector can be safely used, and from which, next or previous, image data should be 
used in the interpolation process. Region la can be interpolated from the next image using 

20 vector D-*. Region lb can be interpolated from the next and the previous image using vector 

D^. Region Ilia can be interpolated from the next and the previous image using vector Df. 

Region Illb can be interpolated from the next image using vector Dn, Two areas still exist, 
i.e. la and Illb, for which no suitable motion vector has been identified. Hereinafter, a 
practical solution for all regions will be introduced. 

25 

So far, we distinguished areas in which covering and uncovering occurs 
between two successive images. However, at the temporal instance where the new image is 
interpolated we need to refine our earlier recognition of covering and uncovering based on 
and €f. To this end, we define 6^' and according to: 

30 

ei (D,, X, n) = [ Y: (Fi^'OiD,, n-1) - Fix ^ (l-a)D,, n)f)/N^j (9) 
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and 

(10) 



where Ngi stands for the number of samples in block B,(X). Instead of a fourth power, a 

second power can also be used. The block B,(X) is not necessarily identical to B{X), A 
5 smaller block size will mostly be preferred, as this allows a more detailed region 
categorization. This error measure has been chosen since it has proven to be the most 
effective, in most cases. 

While we are within region I, it is concluded if e^' <. Th, that we are in 
region lb, and in region la otherwise. The variable Th, here, is an experimentally 

10 determined, small, threshold value. In words: we assume to be in region lb when can be 
considered valid. This occurs when, at the temporal position of the image to be interpolated, 
the two luminance values obtained according to D^, i.e. the one from the delayed field and 
the one from the current field, can be matched with a small error. We assume to be in region 

la when cannot be considered valid. 
15 Similarly, while we are within region III, we conclude that if <, Th, 

that we are in region Ilia, and in region Illb otherwise. In words: we assume to be in region 

Ilia when Df can be considered valid. This occurs when, at the temporal position of the 
image to be interpolated, the luminance values obtained according to Df can be matched with 
a small error. We assume to be in region Illb when Df cannot be considered valid. 

20 

It can be concluded that for any image to be interpolated between the 
previous and the current field, it seems best to interpolate the result from the previous field 
only, or mainly, in case of 'real covering' of the background (region Illb), whereas in case 
of 'real uncovering' of the background (region la), the motion compensated data from the 
25 next field is preferred in the interpolation process. In all other cases, the motion compensated 
interpolator can use the data from both fields. 

So, where a prior art motion compensated average would yield an 
intermediate picmre according to: 
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It) = lf2 {Fix - olD{x, n), n-1) + F{x + (l-a)D(x, n\ «)) (11) 

where o; determines the temporal position of the interpolated image, we now propose to use 
different interpolation strategies for the various regions recognized according to the previous 
section. For example: 

5 

F,{x, n) = 



F{x- 


aDjix, n), n - 1) 




, region lU^ 


Fix* 


il-a)D,(x, n), n) 




, region 




■aDJix, n), n-l)+F(x + (l 


-0L)Dfx, ri), n)) 


, region III^ 




aD^ix, n), n-l)+F(^ + (l 


-a)j5j(^, n), n)) 


, region 




-aD/Jc, n), n-l)+F(Jc-.-(l 


-a)D^(jc, n), «)) 


, otherwise 



Rather then switching between the various interpolation methods in (12), a 
soft switch could be thought of, i.e. a fading between the five alternatives. We will, 
10 however, not further elaborate this thought here. 

In (12), and also in Fig. 2, vectors D, occur, which up to now have no 
meaning. They are required for those parts of the interpolated image where there is either 
covering or uncovering. Therefore it is known that either the previous or the next image 
should be solely used in the interpolation, but no valid vector is available to shift the 
15 corresponding background that either appears or disappears. 

~ - \ In order to solve this problem, we here propose to search for the nearest 
vector describing the background velocity. If we are in the uncovering region I, this nearest 

vector is obtained from the backward predicting motion estimator, generating D^, at a 

location shifted over the negative value of the forward predicting motion vector D^, 

20 multiplied by of, the temporal position of the image to be interpolated. If we are in the 
covering region III, this nearest vector is obtained from the forward predicting motion 
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estimator^ generating D^, at a location shifted over the negative value of the backward 
predicting motion vector D^, multiplied by (1— a). So, more formally: 

_ _ J Dy(x- aDf^ n) , {region z.e. 'real uncovering') 
D^C^y n) + (1 - OL)Dp n) , {region III^ i,e, 'real covering') 



An alternative, prior art, motion compensated interpolation was described 
in [1]. It uses an order statistical filter to make the upcon version more robust with regard to 
erroneous motion vectors. A very basic version was described in [6], and applied in the 
Natural Motion IC [9]. This version uses: 

{X, n) = med {F{x - aD{x, n), Av, F{x + (1 - a)D{x, n), n)) (14) 

with 

Av = ^ (F(Jc, n) ^ F(Jc, n-1)) (15) 
2 

and 

a , {b^a^c V c^a^b) 

15 med (a, c) ^ { b , {a^b^c V c^b^a) (16) 

c , otherwise 



In regions lb and Ilia a safer version of the interpolation according to the 
current invention would use a similar approach. As can be seen from the equations (18) 
below, motion compensated weighted averaging is proposed for 'uncomplicated* areas. In 
20 region lb and llla, i.e. the areas in which there is no acmal covering in the image to be 

interpolated, but yet near a complicated region, the robust interpolation according to (14) is 
proposed. 

In the 'real* covering and uncovering areas, la and lllb, we would like as 
well to propose a robust interpolation method. Thus we again make use of a median filter. 
25 As an input to the median filter, our best guess is the pixel from the previous/current frame 
(depending on covering/uncovering), shifted over Do. Another useful 'candidate pixel' is the 
pixel selected when shifting over the vector that would be valid for the neighboring a/b area 
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(la, in case we are in the uncovering region, Illb otherwise). The third pixel that we propose 
here is the one from the same field shifted over D?.,, where: 



^ ^ ^ I 5y - ^i,, n) , (region i.e. 'real uncovering^ ^^^^ 
J" ' 1 (x + n) , (region III^ i.e. 'real covering') 



or preferably, 



D^^(x, n) = 



- 2 * 

D^(x - (a+ — a)Z)^, n) , (region z.e, 'real uncovering') 

2 ^ (17a) 
(x + ((l-a)+— a)D^, n) , (region III^^ i.e. 'real covering^ 



The thought behind this choice is that D? = in most cases where is 
reliable. In those cases therefore our best guess is actually selected as the output pixel. If D, 
10 and Dnn are not identical, it is likely that confusion exists between region a and b, and the 
chance increases that the second 'candidate pixel' is chosen. Eventually the sequence would 
be processed according to: 



F.(jc, n) = 

med (F(x - aD^(x, n), n-l), F^^, F(x - aD„(x, n), /i-l)) , region III^ 

med \F(x + (l-a)D,(jc, n), n\ F^^, F(x + (l-a)D„(x, n), /i)) , region 

\ med [F(x - aD/Jc, n), /i-l), Av, F(x + (l-a)D/x, n), n)) , region III^ (18) 

mcrf [F(x - aDj(x, n), n-1), Av, F(x + (l-a)D^(5; n), n)) , region 

- (F(x - aD^Jc, n), n-1) + F(x + (l-a)D^(x, n), n)) , otherwise 



where: 

^/na = ^ - '^-D ^ ^ ^ (l-a)^/x, n), n)) (19) 
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and 



= I {Fix - aD^ix, n), n-l) ^ Fix + (l-a)D,(^, n\ n)) (20) 



In order to provide a safer interpolation strategy for the most critical regions (la and Illb) an 
5 other solution has been tested. 

For region l^: 

F.(jc, rt) = med^ (Fix + (1 - a)D^ix, n), n), 
Fix ^ (1 - (t)D^ix, n), n), 

Av, (18a) 
Fix - aD^iXy n), n-l), 

F(jc + (1 - aWx, n), n)) 



10 

and for region 111^: 

F.(J?, n) = med^ [Fix + - (tD^ix, n\ n - 1), 
Fix - aD^^ix, n\ n-l), 

^v, (18b) 
Fix - oiDfx, n), n-l). 

Fix ^ (1 - a)Dfx, n), n)) 

This sophistication experimentally proved to significantly increase the 
interpolation quality. A preferred embodiment of the current invention is shown in the block 
diagram of Fig. 3. An input video signal I is applied to a dual estimator DE having two 
motion vector estimators ME1» ME2. In the dual estimator DE, the input video signal I is 
applied to a field delay FMl, a first input of the first motion vector estimator MEl, and to a 
first input of the second motion vector estimator ME2 thru a shifter S2. An output signal of 
the field delay FMl is applied to a second input of the second motion estimator ME2, and to 
a second input of the first motion vector estimator MEl thru a shifter SI. The first motion 
vector estimator MEl supplies the vector Df, while the second motion vector estimator ME2 



15 



20 
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supplies the vector Db. 

The input video signal I is also applied to a motion-compensated 
interpolator MCI. In the motion-compensated interpolator MCI, the input video signal I is 
applied to a field delay FM2, and to a shifter S3. An output of the field delay FM2 is 

5 applied to a shifter S4. The shifters S3, S4 are controlled by the motion vectors Db, Df 
received from the dual estimator DE. Outputs of the shifters S3, S4 are applied to a median 
circuit med and to an average circuit Av. Outputs of the median circuit med and the average 
circuit Av are applied to a multiplexer MUX which supplies the output signal O to a display 
device CRT for displaying the output signal at, for example, a 100 Hz field rate. The motion 

10 vectors Db, Df, their errors fb, €f, and threshold values ml, m2 are applied to a region 
detector RD which furnishes a control signal to the multiplexer MUX. 

A primary aspect of the invention can be summarized as follows. In 
pictures resulting from motion compensated field rate converters an artifact is visible at the 

15 boundary of moving objects where either covering or uncovering of backgrounds occurs. It is 
the object of the present invention to eliminate this local artifact. To this end, areas where 
covering and uncovering occur are recognized based on the relative prediction quality of two 
motion vectors. The two motion vectors are estimated between the same field pair, but differ 
in that the first is calculated to optimally predict the previous field from the next, whereas 

20 the second tries to predict the next field from the previous. An upconversion method using 
the two motion vectors is also described in this disclosure. 

In a preferred interpolation method, two motion vectors, one predicting 
the next image from the previous image and the other predicting the previous image from the 
next, are taken from both sides of a discontinuity in the vector field, and both applied to 

25 predict the next image from the previous and to predict the previous image from the next, 
and a choice between the vectors is made in dependence on the relative suitability for the 
tested prediction. 

A preferred embodiment can be sununarized as follows. An image 
between neighboring images in a sequence, is interpolated by applying two motion vectors 

30 and V^, one optimally suited for backwards predicting and the other for forwards 

prediction. The embodiment comprises: 

• means to shift data from the next image over a fraction of , 
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• means to shift data from the previous image over a complementary fraction of l^, 

• means to shift data from the previous image over a fraction of V^, 

• means to shift data from the next image over a fraction of . 

• means to combine the shifted data, and possibly other (unshifted) data in dependence on 
5 segmentation information indicating covering/uncovering in the interpolated image, in order 

to form the interpolated image characterized in that in case of covering/uncovering, at least 
one of the vectors that are used in the combination process are taken from a spatial position 
that differs from the spatial position for which it was calculated. Preferably, the position 
from which the vector is taken, is shifted in the opposite direction in which the vector is 
10 pointing with regard to the position for which it was calculated. Advantageously, in case of 

covering at least af^ is used shifted over -aF^, while in case of uncovering at least aK^ is 
used shifted over -aJ^, 

It should be noted that the above-mentioned embodiments illustrate rather 
15 than limit the invention, and that those skilled in the art will be able to design many 

alternative embodiments without departing from the scope of the appended claims. In the 
claims, any reference signs placed between parentheses shall not be construed as limiting the 
claim. The word "comprising" does not exclude the presence of other elements or steps than 
those listed in a claim. The invention can be implemented by means of hardware comprising 
20 several distinct elements, and by means of a suitably programmed computer. In a device 
claim reciting a plurality of different means, several of these means may be embodied by a 
one and the same item of hardware. In an alternative embodiment, the covered and 
uncovered parts are not detected by comparing backward and forward estimation errors, but 
by analyzing direction and magnitude of motion vectors around contours. 
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Claims: 



L A method of detecting covered and uncovered parts in an image (II) to be 

interpolated between neighboring previous (I) and next (III) input images, the method 
comprising the steps of: 

detecting motion from said next input image (III) to said previous input. 
5 image (I), resulting in backward motion vectors (Db) having corresponding backward 
estimation errors (€b); 

detecting motion from said previous input image (I) to said next input 
image (III), resulting in forward motion vectors (Df) having corresponding forward 
estimation errors (6f); 

10 detecting uncovered (I) and covered (III) parts in said neighboring 

previous (I) and next (III) input images, respectively; 

said method being characterized by the further steps of: 
detecting in said thus detected uncovered parts (I), uncovered parts (la) in 
said image (II) to be interpolated by determining second backward estimation errors by 
15 comparing both neighboring previous (I) and next (III) input images when partially shifted 
over said backward motion vectors (Db) to a temporal location of said image (II) to be 
interpolated, and by comparing said second backward estimation errors to a threshold; and 
detecting in said thus detected covered parts (III), covered parts (Illb) in 
said image (II) to be interpolated by determining second forward estimation errors by 
20 comparing both neighboring previous (I) and next (III) input images when partially shifted 
over said forward motion vectors (Df) to a temporal location of said image (II) to be 
interpolated, and by comparing said second forward estimation errors to said threshold. 

2. A method as claimed in claim 1, wherein said step of detecting said 

25 uncovered (I) and said' covered (III) parts in said neighboring previous (I) and next (III) input 
images includes the step of processing said backward estimation errors (e^) and said forward 
estimation errors (e^) to detect said uncovered (I) and said covered (III) parts in said 
neighboring previous (I) and next (III) input images, respectively. 
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3. A device for detecting covered and uncovered pans in an image (II) to be 
interpolated between neighboring previous (I) and next (III) input images, the device 
comprising: 

means for detecting motion from said next input image (III) to said 
5 previous input image (I), resulting in backward motion vectors (D^) having corresponding 
backward estimation errors (6^); 

means for detecting motion from said previous input image (I) to said next, 
input image (III), resulting in forward motion vectors (Df) having corresponding forward 
estimation errors (e^); 

10 means for detecting uncovered (I) and covered (III) parts in said 

neighboring previous (I) and next (III) input images, respectively; 

means for detecting in said thus detected uncovered parts (I), uncovered 

parts (la) in said image (II) to be interpolated by determining second backward estimation 

errors by comparing both neighboring previous (I) and next (III) input images when partially 
15 shifted over said backward motion vectors (D^) to a temporal location of said image (II) to be 

interpolated, and by comparing said second backward estimation errors to a threshold; 

and detecting in said thus detected covered parts (III), covered parts (Illb) in 

said image (II) to be interpolated by determining second forward estimation errors by 

comparing both neighboring previous (I) and next (III) input images when partially shifted 
20 over said forward motion vectors (Df) to a temporal location of said image (II) to be 

interpolated, and by comparing said second forward estimation errors to said threshold. 

4. A device for estimating motion vectors, comprising: 

a device for detecting covered and uncovered parts in an image to be 
25 interpolated as claimed in claim 9; and 

means for furnishing motion vectors for said covered and uncovered parts 
in said image (II) to be interpolated in dependence upon said backward motion vectors (Db) 
and said forward motion vectors (Df). 

30 5. A method of estimating motion vectors for an image (II) to be interpolated 

between neighboring previous (I) and next (III) input images, the method comprising the 
steps of: 

detecting motion from said next input image (III) to said previous input 
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image (I), resulting in backward motion vectors (Pb) having corresponding backward 
estimation errors (6b); 

detecting motion from said previous input image (I) to said next input 
image (III), resulting in forward motion vectors (D^) having corresponding forward 
5 estimation errors (£f); 

detecting uncovered (la) and covered (Illb) parts in said image (II) to be 

interpolated; 

furnishing motion vectors for said uncovered parts (la) in dependence 
upon said forward motion vectors (Df) taken from locations wholly and/or partly shifted over 
10 said backward motion vector (DJ; and 

furnishing motion vectors for said covered parts (Illb) in dependence upon 
said backward motion vectors (Db) taken from locations wholly and/or partly shifted over 
said forward motion vectors (Df). 

15 6. A method as claimed in claim 5, wherein said step of detecting said 

uncovered (la) and said covered (Illb) parts in said image (II) to be interpolated includes the 
step of processing said backward estimation errors {e^ and said forward estimation errors (6f) 
to detect said uncovered (la) and said covered (Illb) parts in said image (II) to be 
interpolated. 

20 

7. A method as claimed in claim 5, wherein said steps of furnishing motion 

vectors for said uncovered (la) and covered (Illb) parts comprise the steps of: 

furnishing first motion vectors for said uncovered parts (la) by taking said 
forward motion vectors (Df) from locations wholly shifted over said backward motion vectors 
25 (Db); 

furnishing second motion vectors for said uncovered parts (la) by taking 
said forward motion vectors (Df) from locations partly shifted over said backward motion 
vectors (Db) correspondmg to a temporal location of said image (II) to be interpolated 
between said neighboring previous (I) and next (III) images; 
30 furnishing first motion vectors for said covered parts (Illb) by taking said 

backward motion vectors (Db) from locations wholly shifted over said forward motion vectors 
(Dr); 

furnishing second motion vectors for said covered parts (Illb) by taking 
said backward motion vectors (Db) from locations partly shifted over said forward motion 
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vectors (Df) corresponding to a temporal location of said image (II) to be interpolated 
between said neighboring previous (I) and next (III) images. 

8'. A method of interpolating an image (II) between neighboring previous (I) 

5 and next (III) input images, the method comprising the steps of: 

estimating motion vectors for said image (II) to be interpolated in 
accordance with claim 5; and 

furnishing said image (II) to be interpolated by means of said backward 
motion vectors (D^), said forward motion vectors (D^), said motion vectors for said 
10 uncovered parts (la), and said motion vectors for said covered parts (Illb). 

9. A device for estimating motion vectors for an image (II) to be interpolated 
between neighboring previous (I) and next (III) input images, the device comprising: 

means for detecting motion from said next input image (III) to said 
15 previous input image (I), resulting in backward motion vectors (DJ having corresponding 
backward estimation errors (et,); 

means for detecting motion from said previous image (I) to said next 
image (III), resulting in forward motion vectors (Df) having corresponding forward 
estimation errors (^f); 

20 means for detecting uncovered (la) and covered (Illb) parts in said image 

(II) to be interpolated; 

means for furnishing motion vectors for said uncovered parts (la) in 

dependence upon said forward motion vectors (Df) taken from locations wholly and/or partly 

shifted over said backward motion vector (D^); and 
25 means for furnishing motion vectors for said covered parts (Illb) in 

dependence upon said backward motion vectors (Dj,) taken from locations wholly and/or 

partly shifted over said forward motion vector (Df). 

10. A display apparatus, comprising: 

30 a device (DE) for estimating motion vectors for an image (II) to be 

interpolated between neighboring previous (I) and next (III) input images as defined in claim 
9; 

means (MCI) for interpolating said image (II) between said neighboring 
previous (I) and next (III) input images by means of said backward motion vectors (D^,), said 
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forward motion vectors (Df), said motion vectors for said uncovered parts, and said motion 
vectors for said covered parts; and 

means (CRT) for displaying an image sequence formed by said 
neighboring previous (I) and next (III) input images and said image (II). 
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